About

This page captures simple demonstrations of some of capabilities of RRAP to Registry Activities, Entities and Actors. If you select the 'Binder' button above, a Jupyter Notebook will be lauched that you can use to further explore these examples. You can change parameters and see results to further expand your understand of RRAP Registry.

With RRAP-IS Registry system can be queried to find all 'Activities', 'Entities' and 'Actors' associated with particular 'Outputs', 'Inputs', or 'Reef Interventions'

Please read documenation ### to understand terms used in this notebook and the future systems

Import general dependencies

import requests
import os
import sys
import json
from json2html import *
from bs4 import BeautifulSoup
import pandas as pd
from IPython.display import IFrame, display, HTML, JSON, Markdown
from mdsisclienttools.auth.TokenManager import DeviceFlowManager

import warnings
warnings.filterwarnings(action='once')

Define global variables, base urls

Note the use structure of the base url, pointing to test servers. If you wish to test your code first, please use the test servers (listed below). Note that these may not retain data and could be down at the time of testing.

Remove testing from the url to use the production urls. Please be careful using production urls, filling the system with test data will require considerable effort to cleanup.

data_api = "https://data-api.testing.rrap-is.com"
registry_api = "https://registry-api.testing.rrap-is.com"
prov_api = "https://prov-api.testing.rrap-is.com"
auth_server = "https://auth.dev.rrap-is.com/auth/realms/rrap"
# garbage = "https://frogs.are.green"
base_urls = {'data_api': data_api, 'registry_api': registry_api, 'prov_api': prov_api, 'auth_server': auth_server}#, 'garbage': garbage}
display('Checking base urls')
# [display(Testing':',value) for key, value in base_urls.items()]
for key, url in base_urls.items():
    try:
        print(f'Testing - {url}', end="")
        r = requests.get(url)
        r.raise_for_status()
        print(f' - Passed')
    except requests.exceptions.HTTPError as err:
        print(f' - Fail')
        raise SystemExit(err)
    except requests.exceptions.RequestException as e:
        # catastrophic error. bail.
        print(f' - Fail')
        raise SystemExit(e)

'Checking base urls'

Testing - https://data-api.testing.rrap-is.com - Passed
Testing - https://registry-api.testing.rrap-is.com - Passed
Testing - https://prov-api.testing.rrap-is.com - Passed
Testing - https://auth.dev.rrap-is.com/auth/realms/rrap - Passed

Setup tokens using device authorisation flow against keycloak server

This could result in a browser window being opened if you don't have valid tokens cached in local storage.

local_token_storage = ".tokens.json"

token_manager = DeviceFlowManager(
    stage="TEST",
    keycloak_endpoint=auth_server,
    local_storage_location=local_token_storage
)

Attempting to generate authorisation tokens.

Looking for existing tokens in local storage.

Validating found tokens

Trying to use found tokens to refresh the access token.

Token refresh successful.

Helper functions

Return to Top

def wrap_html_table(data):
    soup = BeautifulSoup(data)

    ul_tag = soup.find("table")
    div_tag = soup.new_tag("div")
    div_tag['style'] = "width: auto; height: 400px; overflow-y: auto; background-color: #F1F3F4"
    ul_tag.wrap(div_tag)
    new_tag = soup.new_tag("details")
    div_tag.wrap(new_tag)
    
    tag = soup.new_tag("summary")
    tag.string = "Results"
    soup.div.insert_after(tag)

    return soup.prettify()

Endpoint Documentation

Endpoint documentation can be found by appending either /docs or /redoc on the end a base URL.

For example:

Then select from the menu an endpoint function call e.g. /check-access/check-general-access

Then append the function call onto the base url e.g. https://registry-api.testing.rrap-is.com/check-access/check-general-access

Return to Top

Demonstrations

Dataset Registry APIs

Dataset Registry - Redoc FastAPI

List all datasets

Return to Top

auth = token_manager.get_auth

# Setup postfix and endpoint
postfix = "/registry/items/list-all-datasets"
endpoint = data_api + postfix

# When making the request, use auth=auth() - this will ensure tokens are valid
# right at the point of using them, including potentially auto refreshing!
response = requests.get(endpoint, auth=auth())

json_obj_in_html = json2html.convert( response.json()['registry_items'])
HTML( wrap_html_table(json_obj_in_html))

Model Registry APIs

List all registered models

Return to Top

auth = token_manager.get_auth

# Setup postfix and endpoint
postfix = "/registry/entity/model/list"
endpoint = registry_api + postfix

# When making the request, use auth=auth() - this will ensure tokens are valid
# right at the point of using them, including potentially auto refreshing!
response = requests.get(endpoint, auth=auth())
saved_model_id = response.json()['items'][0]['id']
json_obj_in_html = json2html.convert( response.json())
HTML( wrap_html_table(json_obj_in_html))

Fetch registered model

From the above output copy the handle id and use it to fetch

Return to Top

auth = token_manager.get_auth

# Setup postfix and endpoint
postfix = "/registry/entity/model/fetch"

params = {"id" : saved_model_id}
endpoint = registry_api + postfix

# When making the request, use auth=auth() - this will ensure tokens are valid
# right at the point of using them, including potentially auto refreshing!
response = requests.get(endpoint, params=params, auth=auth())

json_obj_in_html = json2html.convert( response.json())
HTML( wrap_html_table(json_obj_in_html))

Register a model

auth = token_manager.get_auth

# Setup postfix and endpoint
postfix = "/registry/entity/model/create"
payload = {
  "display_name": "RRAP-model (repo commit id - 1234)",
  "name": "RRAP-model-commit-1234",
  "description": "Example description string",
  "documentation_url": "https://bitbucket.csiro.au/projects/MAE/repos/pybrat/readme.md",
  "source_url": "https://bitbucket.csiro.au/projects/MAE/repos/pybrat/commits/824fc954c7e85c8b4b94044fc22cdbf83e365735"
}
endpoint = registry_api + postfix 

# When making the request, use auth=auth() - this will ensure tokens are valid
# right at the point of using them, including potentially auto refreshing!
response = requests.post(endpoint, json=payload, auth=auth())

json_obj_in_html = json2html.convert( response.json())
HTML( wrap_html_table(json_obj_in_html))

Delete a registered model

auth = token_manager.get_auth

# Setup postfix and endpoint
postfix = "/registry/entity/model/delete"
params = { 'id': '10378.1/1687609' }
endpoint = registry_api + postfix

# When making the request, use auth=auth() - this will ensure tokens are valid
# right at the point of using them, including potentially auto refreshing!
response = requests.delete(endpoint, params=params, auth=auth())


json_obj_in_html = json2html.convert( response.json())
HTML( wrap_html_table(json_obj_in_html))

Dataset Template Registry APIs

Return to Top

auth = token_manager.get_auth

# Setup postfix and endpoint
postfix = "/registry/entity/dataset_template/create"
payload = {
  "display_name": "string",
  "usage_information": {
    "usage_type": "PARAMETER_FILE",
    "usage_description": "string"
  },
  "structural_template": {
    "additional_description": "string",
    "file_information": [{"string":"value"},{"bees":"knees"}]
  },
  "spatial_information": {},
  # "temporal_information": {
  #   "start_time": "2022-09-01T00:34:12Z",
  #   "end_time": "2022-09-02T00:34:12Z",
  #   "interval": 1}
}

endpoint = registry_api + postfix 

# When making the request, use auth=auth() - this will ensure tokens are valid
# right at the point of using them, including potentially auto refreshing!
response = requests.post(endpoint, json=payload, auth=auth())

json_obj_in_html = json2html.convert( response.json())
HTML( wrap_html_table(json_obj_in_html))

List all Dataset Templates

auth = token_manager.get_auth

# Setup postfix and endpoint
postfix = "/registry/entity/dataset_template/list"

endpoint = registry_api + postfix

# When making the request, use auth=auth() - this will ensure tokens are valid
# right at the point of using them, including potentially auto refreshing!
response = requests.get(endpoint, auth=auth())

json_obj_in_html = json2html.convert_json_node( response.json())
HTML( wrap_html_table(json_obj_in_html))

Fetch a specific Dataset Template

auth = token_manager.get_auth

# Setup postfix and endpoint
postfix = "/registry/entity/dataset_template/fetch"
params = "?id=10378.1/1690432"
endpoint = registry_api + postfix + params

# When making the request, use auth=auth() - this will ensure tokens are valid
# right at the point of using them, including potentially auto refreshing!
response = requests.get(endpoint, auth=auth())

json_obj_in_html = json2html.convert_json_node( response.json())
HTML( wrap_html_table(json_obj_in_html))

Agent Registry APIs

Create a Agent

Return to Top

auth = token_manager.get_auth

# Setup postfix and endpoint
postfix = "/registry/agent/person/create"
payload = { 
    "display_name" : "Reef Modeller for eReef",
    "first_name" : "John",
    "last_name" : "Doe",
    "email" : "John.Doe@uni-of.au",
    "orcid" : "https://orcid.org/0000-0000-000-0000"
}
endpoint = registry_api + postfix 

# When making the request, use auth=auth() - this will ensure tokens are valid
# right at the point of using them, including potentially auto refreshing!
response = requests.post(endpoint, json=payload, auth=auth())

#For display only
json_obj_in_html = json2html.convert_json_node( response.json())
HTML( wrap_html_table(json_obj_in_html))

List all Agents

auth = token_manager.get_auth

# Setup postfix and endpoint
postfix = "/registry/agent/person/list"
endpoint = registry_api + postfix

# When making the request, use auth=auth() - this will ensure tokens are valid
# right at the point of using them, including potentially auto refreshing!
response = requests.get(endpoint, auth=auth())

json_obj_in_html = json2html.convert_json_node( response.json())
HTML( wrap_html_table(json_obj_in_html))

Organistation Registry APIs

Create a Organisation

Return to Top

auth = token_manager.get_auth

# Setup postfix and endpoint
postfix = "/registry/agent/organisation/create"
payload = { 
    "display_name" : "CSIRO",
    "name" : "Commonwealth Scientific and Industrial Research Organisation",
    "ror" : "https://ror.org/03qn8fb07"
}
endpoint = registry_api + postfix 

# When making the request, use auth=auth() - this will ensure tokens are valid
# right at the point of using them, including potentially auto refreshing!
response = requests.post(endpoint, json=payload, auth=auth())

json_obj_in_html = json2html.convert_json_node( response.json())
HTML( wrap_html_table(json_obj_in_html))